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To  a  large  degree,  the  classical  approach  to  problem  solving  In 
operations  research  (OR)  Is  to  fit  a  real  life  situation  Into  a  well- 
known  OR  model.  When  OR  models  are  used  to  deal  with  major  policy 
problems  In  which  the  underlying  processes  are  not  well  understood, 
this  effort  results  In  too  much  simplification. 

Due  to  an  inability  to  perceive  all  uncertainties,  and  a  consequent 
wish  to  retain  flexibility  once  the  decisions  are  made,  decisionmakers 
are  more  Interested  in  the  "robustness"  of  their  policy  decisions  than 
their  "optimality,"  which  becomes  a  vague  concept  due  to  the  nature  of 
these  problems. 

This  paper  emphasizes  the  desirability  of  robustness  and  criticizes 


attempts  to  fit  an  operationalized  measure  of  robustness  Into  an  opti¬ 
mization  structure,  by  the  aid  of  a  decision  analytic  example. 


I.  INTRODUCTION 


The  role  of  operations  research  (OR)  models  In  public-policy 
decisionmaking  have  been  viewed  in  two  philosophically  different  ways: 
(1)  As  tools  to  be  utilized  to  obtain  the  optimal  solution  to  a  policy 
problem,  and  (2)  as  a  means  of  providing  insights  into  a  problem  area. 
Illuminating  the  inherent  trade-offs  in  alternative  solutions,  screen¬ 
ing  alternatives,  and  generating  alternatives  for  further  exploration. 
The  virtues  of  the  second  view,  to  which  I  subscribe,  have  been  des¬ 
cribed  elsewhere  (for  example,  II]  and  [5]). 

Some  who  have  recognized  the  drawbacks  of  using  models  to  obtain 
an  "optimal"  solution  (some  of  the  drawbacks  are  detailed  below)  have 
suggested  using  robustness  as  a  remedy.  However,  in  operationalizing 
the  idea  they  have  tried  to  quantify  robustness.  This  paper  criticizes 
attempts  to  calculate  robustness  values  and  argues  for  other  operation¬ 
alizations  of  the  idea,  in  view  of  recent  trends  in  OR. 

II.  SPECTRUM  OF  PROBLEMS 

OR  models  have  been  applied  to  a  wide  range  of  problems  since  the 
1950s.  Although  vastly  different  in  nature  they  can  be  characterized 
by  two  groups  which  span  the  range,  say  Type  A  (well  defined)  and 
Type  Z  (fuzzy). 

Type  A  Problems 

These  usually  have  a  widely  accepted  single  objective,  or  an 
objective  which  can  serve  as  a  good  proxy  for  all  the  others.  Alter¬ 
natives  are  easy  to  identify.  There  is  little  room  for  unperceived  or 
unquantified  uncertainties,  so  risks  can  usually  be  Incorporated  into 
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a  model  using  probabilities.  Models  built  for  these  problems,  more 
often  than  not,  represent  the  real  situation  closely.  Type  A  problems 
are  generally  tactical  In  nature.  Decisions,  once  taken,  are  applied 
repeatedly  to  the  situation.  Hence,  expected  value  can  be  used  as  the 
relevant  measure  of  payoff.  OR  models  have  found  wide  application  to 
this  class  of  problems  (e.g.,  determining  Input-output  mix  for 
refineries) . 

Type  Z  Problems 

These  problems  have  a  multi-objective  nature  and  generally  the 
weights  to  be  given  to  different  objectives  are  highly  subjective. 
Therefore,  multiple  objectives  cannot  be  evaluated  by  a  single,  uni¬ 
versally-accepted  criterion.  Some  Issues  are  qualitative  and  it  may 
be  very  difficult  to  Introduce  them  into  the  model  in  a  quantitative 
manner.  For  this  reason,  many  relevant  Issues  may  remain  unmodelled. 
Uncertainties  are  extensive,  both  because  of  the  nature  of  the  problem 
and  because  of  the  unmodelled  Issues.  Furthermore,  some  uncertainties 
cannot  be  expressed  In  terms  of  probabilities,  mainly  because  they 
cannot  be  fully  perceived  ahead  of  time.  Finally,  Type  Z  problems 
involve  decisions  which  are  generally  strategic  in  nature.  Their 
reversibility  Is  very  limited  and  costly,  and  their  consequences  are 
long-standing.  Therefore,  declfiionmakers  are  more  interested  in  the 
robustness  of  the  solutions  than  In  their  expected  payoffs.  Type  Z 
problems  can  be  characterized  by  large  Investment  decisions  and  major 
public  policy  issues. 
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Applicatlon  of  OR  models,  and  for  that  matter  any  model,  to  this 
type  of  problem  has  raised  major  concerns  with  decisionmakers  and  some 
policy  analysts. 

III.  ROBUSTNESS  TO  THE  RESCUE 

Being  aware  of  the  difficulties  of  utilizing  OR  models  to  find  an 
"optimal"  solution  to  Type  Z  problems,  some  OR  practitioners  have 
suggested  using  robustness  as  a  relative  criterion 

Robustness  Is  a  characteristic  of  an  alternative.  It  may  be 
defined  as  the  Insensitivity  to  the  assumptions  of  a  model,  and  to  the 
value  judgements  placed  on  different  objectives  of  a  problem. 

Therefore,  alternatives  which  will  perform  well  under  many  different 
states  of  nature  and  on  most  or  all  the  dimensions  under  consideration 
are  considered  robust.  Decisionmakers  are  increasingly  more  interested 
in  "robust  solutions"  rather  than  "optimal"  ones  as  they  are  involved 
more  with  problems  near  the  Type  Z  end  of  the  spectrum. 

IV.  ROBUSTNESS  MEASURES  AND  THEIR  DRAWBACKS 

The  recognition  that  solutions  may  be  robust  with  respect  to  only 
some  of  the  assumptions.  Illuminates  the  difficulty  of  coming  up  with 
a  general  measure  of  robustness.  Nevertheless,  a  number  of  papers 
written  on  the  subject  have  attempted  to  measure  robustness  (e.g.,  [3], 
[6],  [7],  [8],  [9],  [10],  [12]). 
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In  the  following,  I  use  one  of  the  robustness  measures  mentioned  In 
the  literature  as  an  example  to  Illustrate  some  of  the  drawbacks  of 
these  measures,  namely  their  sensitivity  to  the  definition  of  "satis¬ 
factory  outcomes,"  the  number  of  alternatives  considered,  and  unbalanced 
payoff  values. 

Rosenhead,  et  al.  [7]  view  robustness  as  retaining  flexibility  In 
the  Initial  moves  of  a  multi-stage  decisionmaking  process.  In  their 
words,  "a  plan  whose  Initial  decisions  limit  the  future  as  little  as 
possible  has  an  evolutlonalry  advantage  In  an  uncertain  world."  They 
define  a  measure  of  robustness,  r^  for  a  decision,  d^  as 


r 


1 


n(s^) 

n(s) 


where  n(s)  is  the  number  of  satisfactory  outcomes  available  and  n(s^) 
is  the  nunber  of  satisfactory  outcomes  attainable  after  the  decision, 
d^.  Let's  apply  this  definition  to  the  following  example  to  show  some 
of  the  weak  points. 

Example  2. 

The  California  government  is  interested  in  whether  or  not  to 

ic 

create  incentives  to  promote  crude  oil  production;  to  do  so,  $5  will 


ie 

All  figures  are  in  billions  of  dollars  and  represent  present 
value  of  actual  amounts  that  will  be  spent  or  earned  In  the  next  15 
years.  The  numbers  are  hypothetical. 


have  Co  be  invested  in  equipment  for  a  complex  but  more  efficient  oil 
production  technology.  If  these  investments  are  undertaken,  production 
costs  will  decrease  for  medium  and  high  production  levels  from  $17  to 
$10  and  from  $31  to  $19,  respectively,  but  for  low  production  levels 
operating  costs  will  Increase  by  $2  from  the  previous  $5  level,  due 
to  the  more  complex  technology.  Crude  oil  demand  will  be  met  by  a 
combination  of  In-staCe  production  and  Imports.  The  three  levels  of 
Imports  being  considered:  low,  medium,  and  high;  are  expected  to  cost 
$2,  $11,  and  $20,  respectively.  If  demand  turns  out  to  be  low,  it  can 
be  met  separately  by  a  medium  level  of  Imports,  or  by  medium  production, 

or  jointly  by  low  import  and  low  production.  Crude  oil  in  the  case  of 

* 

low  demand  will  have  a  value  of  $16.  If  demand  turns  out  to  be  high, 
crude  oil  will  be  valued  at  $45  and  the  demand  can  be  met  either  with 
high  imports  and  medium  production  or  with  medium  imports  and  high 
production,  a  and  b  are  the  probabilities  of  low  and  high  demand, 
respectively.  This  information  can  be  displayed  on  a  decision  tree, 
omitting  outcomes  with  negative  payoffs. 


a 

The  value  of  crude  oil  is  assumed  to  be  unaffected  by  production 
decisions. 


FIGURE  I 


denotes  a  decision  node  and  O  denotes  a  chance  node. 


Sensitivity  to  the  Definition  of  "Satisfactor 
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FIGURE  I A  (Same  as  Fig.  I) 
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Let  a  cutoff  value  be  defined  as  the  minimum  payoff  value  which  the 
decisionmakers  will  accept  as  satisfactory. 

If  the  cutoff  point  for  a  satlsfa-'  ory  outcome  is  chosen  to  be 
eight,  then  "  ^2  ^  hence  robustness  of  and  d^  are  equal, 
but  the  choice  of  a  cutoff  point  is  arbitrary  and  A  can  Influence 


the  measure  in  undesirable  ways.  For  example,  if  the  cutoff  noint 
was  chosen  to  be  nine,  in  Figure  lA,  r^  would  be  greater  than  r^ 

=  2/3  ^  ^2  ~  i/3).  Graph  I,  which  shows  the  preferred  alterna¬ 
tives  versus  cutoff  values,  may  be  Informative  and  may  help  for  the 
choice  of  a  cutoff  point. 

Graph  I 


2  4  6  8 

Cutoff  values 


If  1  <  cutoff  _<  5,  d^  is  better  than  d^  with  respect  to  robustness. 
If  8  <  cutoff  £  10,  dj^  is  better  with  respect  to  robustness.  For  other 
cutoff  values,  robustness  does  not  discriminate  between  the  alternatives 
in  this  exan^le. 

Sensitivity  to  Assumed  Payoff  Values 

One  way  to  get  around  the  above  problem  would  be  to  use  the  values 
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o£  the  payoffs  rather  than  the  zero-one  coding  assigned  to  each  outcome 
(by  Its  magnitude  w.r.t.  the  cutoff  point).  Such  a  measure  will  give 
the  following  robustness  scores,  rr^  for  decisions  In  Ex.  2. 

23  25 

^^1  48  ’^’^2  '  48 

Note  that  measure  rr^  assumes  equal  probabilities  for  a  and  b  (a 
reasonable  assumption  In  case  of  absolute  Ignorance). 

Another  way  may  be  the  comparison  of  the  lengths  of  the  regions 
of  cutoff  points  where  one  decision  Is  preferred  over  the  other.  (See 
Graph  I.)  The  measure,  1^  ,  will  give  the  following  scores  for  Ex.  2, 

lj^  =  10-8  =  2  I2  ==5-1  =  4 

This  wo\ild  Imply  that  d^  Is  preferable. 

In  still  a  different  version,  Pye  [4]  incorporates  the  ideas  of 
values  and  satisfactory  outcomes  in  his  last  measure  of  robustness  by 
assigning  a  value  of  zero  to  unsatisfactory  outcomes  and  using  the 
actual  payoff  values  for  satisfactory  outcomes. 

ITie  problem  with  using  values  is  that  a  very  large  value  will 
distort  the  measures.  For  example,  in  Ex.  2,  if  one  of  the  payoffs, 
say  10,  were  1000  Instead,  it  would  have  changed  the  robustness  measures 
mentioned  In  this  section  in  favor  of  d^  even  if  the  probability  of  b 
were  very  low  (i.e.,  lO”®) . 

Also,  when  the  payoffs  are  multi-attribute  in  nature,  reducing 
them  to  a  single  value  may  be  very  hard  and  controversial,  but 
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classlfylng  the  alternatives  as  satisfactory  and  unsatisfactory  may 
be  much  easier. 


Insensitivity  to  the  Location  of  Retained  Flexibility 

The  next  example  is  illustrative  of  the  point  that  Rosenhead's 
measure  r^  is  insensitive  to  the  location  of  the  satisfactory  outcomes 
after  the  decision  between  d^  and  d2» 


Example  2a. 

This  example  is  the  same  as  the  previous  one,  except  that  an  alter¬ 
native  is  added:  If  Investments  are  undertaken,  production  levels  may  be 
Increased  to  a  very  high  level  (v.h.),  dg,  at  a  cost  of  $31.  Then  the 
decision  tree  will  look  as  follows: 


FIGURE  II 
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The  measure  does  not  account  for  the  fact  that  all  three  of  the 


satisfactory  outcomes  after  lie  on  branch  b,  l.e.,  can  be 
exercised  only  if  nature  picks  b.  In  contrast,  there  is  a  satisfactory 
outcome  no  matter  what  nature  picks  if  the  initial  decision  were  d^. 
Therefore,  r^  misleads  us  in  terms  of  robustness. 

Sensitivity  to  the  Number  of  Alternatives  Considered 

Example  2  can  be  elaborated  by  including  a  medium  demand  level 
scenario.  Let  d,  e,  and  f  be  the  probabilities  of  high,  medium,  and 
low  demand,  respectively.  Let's  assume  that  medium  demand  can  be  met 
either  with  medium  production  and  medium  imports,  d^,  or  with  high 
production  and  low  imports,  The  value  of  crude  oil  will  be  $36 

in  this  case.  Therefore,  if  investments  were  undertaken,  d^  and  ‘*10 

would  cost  (10  +  11  +  5=)  $26  and  (19  +  2  +  5=)  $26.  Otherwise,  d^ 
would  cost  (17  +  11=)  $28  and  d^^  would  cost  (31  +  2=)  $33.  Figure 
III  summarizes  the  new  situation.  When  e  +  f  =  b,  this  example  is 
equivalent  to  Ex.  2,  but  the  measure  r^^  gives  significantly  different 
answers  with  these  two  different  ways  of  grouping  alternatives.  When 
e  +  f  is  marginally  different  from  b,  which  model  to  choose  remains 
as  a  subjective  decision  to  the  analyst. 

Therefore,  the  robustness  measures  mentioned  in  the  literature 
themselves  are  not  robust  with  respect  to  the  definition  of  satisfac¬ 
tory  outcomes,  number  of  alternatives  considered,  and  unbalanced  payoff 
values.  Clearly,  measuring  robustness  is  by  definition  a  futile 
effort  since  it  requires  knowledge  of  unperceived  uncertainties. 


-13- 


Furthermore,  using  a  formula  to  calculate  robustness  values  and  using 
these  values  for  comparison  purposes,  is  nothing  more  than  optimizing 
with  respect  to  a  new  criterion  (i.e.,  "robustness"  replaces  "opti¬ 
mization").  Hence,  it  is  open  to  the  same  criticisms  which  lead  the 
way  to  searching  robust  solutions. 

V.  ARE  WE  OUT  OF  HOPE? 

Inability  to  measure  robustness  does  not  imply  inability  to 
operationalize  the  idea.  In  fact,  some  recent  trends,  which  originated 
from  the  viewpoint  of  using  OR  models  as  tools  to  gain  insists  rather 
than  to  find  the  "optimal  solution",  are  towards  operationalizing  the 
idea  of  robustness. 

For  example,  displaying  information  about  each  alternative  on  each 
dimension  of  the  problem  to  clarify  the  inherent  trade-offs  (  e.g.,  the 
"scorecard"  method  deArlsed  by  Goeller  et  al.  f4];  or  [5])  is  such  an 
effort.  This  is  valuable  because  it  is  very  difficult  and  in  certain 
cases  practically  Impossible  (i.e.,  when  there  is  a  large  group  of 
decisionmakers  who  cannot  agree  on  the  relative  merits  of  different 
objectives)  to  reduce  the  dimensions  under  consideration  into  one. 

Also,  communicating  the  insights  gained  through  modelling  effort  to  the 
decisionmakers  by  displaying  information,  will  leave  the  decision  to 
the  decisionmakers  and  allow  them  to  incorporate  hidden,  unmodelled 
issues  into  their  decisions. [11]  Where  some  Issues  cannot  be  modelled 
quantitatively,  a  qualitative  analysis  of  those  issues  will  be  of  more 
use  to  the  decisionmakers  than  studies  which  ignore  them. 

Greater  utilization  of  recent  advances  in  computer  technology  by 
user-interactive  computer  modelling  may  enhance  analysts'  capabilities 
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to  respond  Co  decisionmakers'  informational  needs  [13]. 

In  a  paper  yet  to  be  published.  Brill,  et  al.  [2]  address 
the  issue  of  devising  methods  to  generate  alternatives  (the  Hop-Skip- 
Jump  method) .  Throughout  this  paper  I  have  elaborated  on  the  need  for 
finding  satisfactory  robust  solutions  rather  than  optimal  ones. 

Therefore,  one  needs  to  evaluate  many  alternatives  which  have  satisfac¬ 
tory  outcomes  (if  there  isn't  any,  one  probably  has  to  reduce  his 
expectations);  Brill's  paper  deals  with  identifying  different  alternatives. 


VI.  CONCLUSIONS 

OR  models  have  been  and  will  be  useful  to  Che  decisionmakers. 

However,  model  building  and  alternative  identification  are  both  Influenced 
by  analysts'  subjective  decisions,  and  their  choices  may  have  major 
implications  on  Che  conclusions  of  an  analysis.  Constraints  Imposed 
by  the  way  analysis  is  conducted,  when  not  explicitly  stated,  may  reduce 
the  decisionmaker's  confidence  in  Che  study,  hence  diminish  the  usefulness 
of  the  analysis  as  an  aid  to  decisionmaking.  Therefore,  all  limitations 
Imposed  on  the  problem  by  the  modelling  efforts  should  be  made  explicit. 

In  real  life  problems  it  is  very  hard  to  find  dominating  or  even 
robust  solutions  with  respect  to  all  the  assumptions  and  Impacts.  Even 
then  alternatives  that  are  robust  with  respect  to  only  some  of  the 
assumptions  may  still  be  worth  taking  by  the  decisionmakers.  Although 
robustness  of  an  alternative  is  important  to  consider,  it  should  not 
be  used  as  Che  sole  criterion  to  evaluate  alternatives  especially  since 
robustness  cannot  be  reduced  to  a  number.  Therefore,  robustness  of 
different  alternatives  should  be  identified,  possibly  by  using  sensitivity 
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analysls,  and  discussed.  Analyzing  many  alternatives  and  communicating 
the  Insights  gained  to  the  decisionmakers  will  receive  better  acceptance 
than  prescribed  "optimal  solutions"  which  are  too  sensitive  to  the 
assumptions  to  be  applicable. 

i 

i 

I 
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